kola/kernel-replace: adapt for RHCOS and SCOS #3048

c4rt0 · 2024-07-02T11:27:44Z

There were instances where kola/switch-kernel test would fail.
It was recommended to remove it (PR) and generalize this test, so that it can work with FCOS, RHCOS and SCOS.

jbtrystram

Shellcheck will complain other than this it looks good.
Could you run the test locally on RHCOS ?

tests/kola/rpm-ostree/kernel-replace

c4rt0 · 2024-07-05T18:40:46Z

Could you run the test locally on RHCOS ?

Just tried it now, it's failing. I'm now investigating:

=== RUN   ext.config.shared.rpm-ostree.kernel-replace
systemctl status kola-runext.service:
× kola-runext.service
     Loaded: loaded (/etc/systemd/system/kola-runext.service; static)
     Active: failed (Result: exit-code) since Fri 2024-07-05 18:34:49 UTC; 399ms ago
   Duration: 36.717s
    Process: 1266 ExecStart=/usr/local/bin/kola-runext-kernel-replace (code=exited, status=1/FAILURE)
   Main PID: 1266 (code=exited, status=1/FAILURE)
        CPU: 33.418s

Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1270]: ++ grep '^ID=' /etc/os-release
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + OS_ID=rhcos
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + image_dir=/var/tmp/coreos
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + image=oci:/var/tmp/coreos
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + image_pull=ostree-unverified-image:oci:/var/tmp/coreos
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + tmp_imagedir=/var/tmp/coreos-tmp
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1273]: ++ arch
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + arch=x86_64
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + kver=6.2.9-300.fc38.x86_64
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + case "${AUTOPKGTEST_REBOOT_MARK:-}" in
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + skopeo copy oci:/var/tmp/coreos containers-storage:localhost/rhcos
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1274]: time="2024-07-05T18:34:13Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1274]: Getting image source signatures
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1274]: Copying blob sha256:5d033c3086bc3b8a75fe007c50fa48d18cdea48560bd6d6b2b359d79397bb167
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1274]: Copying config sha256:5b8ccce2c709b5c14f75df3761da4eccfcef89d39165711f14c71196df75b28c
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1274]: Writing manifest to image destination
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + rm /var/tmp/coreos -rf
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1307]: ++ mktemp -d
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + td=/tmp/tmp.q2EfPgCVdW
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + cd /tmp/tmp.q2EfPgCVdW
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1310]: ++ grep Version
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1309]: ++ rpm-ostree --version
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + version=' Version: '\''2024.3'\'''
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + cat
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + touched_resolv_conf=0
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + test '!' -f /etc/resolv.conf
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + podman build --net=host -t localhost/rhcos-derived --squash .
Jul 05 18:34:48 qemu0 podman[1313]: 2024-07-05 18:34:48.253248775 +0000 UTC m=+0.050040248 system refresh
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1313]: STEP 1/3: FROM localhost/rhcos
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1313]: STEP 2/3: RUN rpm-ostree cliwrap install-to-root /
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1345]: Successfully enabled cliwrap for /
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1313]: STEP 3/3: RUN rpm-ostree override replace     https://koji.fedoraproject.org/koji/buildinfo?buildID=2178613 &&     ostree container commit
Jul 05 18:34:49 qemu0 kola-runext-kernel-replace[1397]: Downloading https://koji.fedoraproject.org/koji/buildinfo?buildID=2178613...done
Jul 05 18:34:49 qemu0 kola-runext-kernel-replace[1397]: (rpm-ostree override replace:2): libdnf-WARNING **: 18:34:49.701: failed to read RPM: /tmp/.tmpysF0mq/13.rpm: not a rpm, skipping
Jul 05 18:34:49 qemu0 kola-runext-kernel-replace[1397]: error: Invalid RPM file: /tmp/.tmpysF0mq/13.rpm
Jul 05 18:34:49 qemu0 podman[1313]: 2024-07-05 18:34:49.787637333 +0000 UTC m=+1.584428810 image build  
Jul 05 18:34:49 qemu0 kola-runext-kernel-replace[1313]: Error: building at STEP "RUN rpm-ostree override replace     https://koji.fedoraproject.org/koji/buildinfo?buildID=2178613 &&     ostree container commit": while running runtime: exit status 1
Jul 05 18:34:49 qemu0 systemd[1]: kola-runext.service: Main process exited, code=exited, status=1/FAILURE
Jul 05 18:34:49 qemu0 systemd[1]: kola-runext.service: Failed with result 'exit-code'.
Jul 05 18:34:49 qemu0 systemd[1]: kola-runext.service: Consumed 33.418s CPU time.
--- FAIL: ext.config.shared.rpm-ostree.kernel-replace (171.99s)
        cluster.go:162: Error: Unit kola-runext.service exited with code 1
        cluster.go:162: 2024-07-05T18:34:50Z cli: Unit kola-runext.service exited with code 1
        harness.go:1265: kolet failed: : kolet run-test-unit failed: Process exited with status 1
FAIL, output in tmp/kola
Error: harness: test suite failed

tests/kola/rpm-ostree/kernel-replace

jlebon · 2024-07-05T18:47:05Z

Trying to replace the RHCOS kernel with a Fedora one in theory should work, but I think we can get closer to what users are likely to do by replacing with a CentOS Stream kernel instead. So basically:

FCOS: replace with Fedora kernel
RHCOS and SCOS: replace with CentOS Stream kernel

For Fedora, the test currently pins to a specific Koji build, which is stale now. We should bump it. Once we have dnf in all the streams, we should probably change that actually to just dynamically always take the latest kernel that's != to the kernel embedded in the container image (similarly for RHCOS/SCOS too).

c4rt0 · 2024-07-09T16:12:02Z

I ran this on both rhcos (4.17) and the latest fcos build. Test passes as expected.

c4rt0 · 2024-07-09T16:18:11Z

Trying to replace the RHCOS kernel with a Fedora one in theory should work, but I think we can get closer to what users are likely to do by replacing with a CentOS Stream kernel instead. So basically:
* FCOS: replace with Fedora kernel

* RHCOS and SCOS: replace with CentOS Stream kernel

That's now done.

For Fedora, the test currently pins to a specific Koji build, which is stale now. We should bump it.

That's also done.

Once we have dnf in all the streams, we should probably change that actually to just dynamically always take the latest kernel that's != to the kernel embedded in the container image (similarly for RHCOS/SCOS too).

I can look into this in the future.

tests/kola/rpm-ostree/kernel-replace

c4rt0 · 2024-07-10T12:01:51Z

Again, latest changes successfully passed test both on fcos and rhcos. Thank you for all of the suggestions!

Completed.

jlebon

Suggested commit title:

kola/kernel-replace: adapt for RHCOS and SCOS

For the commit message, I think in general we should avoid just linking to some other URL that has all the context. Let's inline that context, and provide the URL only as an additional reference in the footer.

tests/kola/rpm-ostree/kernel-replace

jlebon

LGTM overall. Thoughts on suggested commit title in #3048 (review) ? I think mentioning OS_ID there isn't really capturing it.

tests/kola/rpm-ostree/kernel-replace

jbtrystram

LGTM overall. I am a bit concerned by the hard-coded kernel package URLs, won't they go stale eventually, breaking the test?

I don't have a good alternative to suggest though, except adding an item to the rebase checklist..

tests/kola/rpm-ostree/kernel-replace

c4rt0 · 2024-07-16T11:15:54Z

LGTM overall. I am a bit concerned by the hard-coded kernel package URLs, won't they go stale eventually, breaking the test?

I don't have a good alternative to suggest though, except adding an item to the rebase checklist..

I agree, @jlebon also talked about it here: #3048 (comment). Let's wait with that until we have DNF in all of the streams.

@jlebon

In one of the older issues, kola switch-kernel test was failing. In the discussion @jlebon suggested to remove named test and make it external, so that it can be utilized by FCOS, and also SCOS/RHCOS. Additionally, the kernel version for FCOS was updated in this PR to the latest stable version `6.9.8-200.fc40`. See: coreos/coreos-assembler#1245

c4rt0 requested a review from jbtrystram July 2, 2024 11:28

jbtrystram previously requested changes Jul 2, 2024

View reviewed changes

tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved

tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved

jlebon reviewed Jul 4, 2024

View reviewed changes

tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved

c4rt0 force-pushed the PR/kernel branch 2 times, most recently from d95cceb to 18d9d86 Compare July 5, 2024 18:37

jlebon reviewed Jul 5, 2024

View reviewed changes

tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved

c4rt0 force-pushed the PR/kernel branch 2 times, most recently from 1899c9f to 3ec17ba Compare July 9, 2024 16:08

c4rt0 requested review from jbtrystram and jlebon July 9, 2024 16:20

jlebon reviewed Jul 10, 2024

View reviewed changes

c4rt0 force-pushed the PR/kernel branch from 3ec17ba to 241ed7c Compare July 10, 2024 11:56

c4rt0 force-pushed the PR/kernel branch from 241ed7c to 232f923 Compare July 10, 2024 12:10

c4rt0 marked this pull request as ready for review July 10, 2024 12:11

c4rt0 force-pushed the PR/kernel branch 2 times, most recently from e99b318 to e4c36a4 Compare July 10, 2024 12:24

c4rt0 force-pushed the PR/kernel branch from e4c36a4 to de0588a Compare July 10, 2024 14:11

c4rt0 requested a review from jlebon July 10, 2024 15:36

jlebon reviewed Jul 10, 2024

View reviewed changes

tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved

tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved

tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved

c4rt0 force-pushed the PR/kernel branch from de0588a to 691d91b Compare July 15, 2024 12:49

c4rt0 requested a review from jlebon July 15, 2024 12:50

c4rt0 force-pushed the PR/kernel branch 2 times, most recently from 5650a3a to 03d962c Compare July 15, 2024 13:03

jlebon mentioned this pull request Jul 15, 2024

kola/switch-kernel: rpm-ostree fails to switch from Default to RT Kernel coreos/coreos-assembler#1245

Closed

jlebon reviewed Jul 15, 2024

View reviewed changes

tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved

c4rt0 changed the title ~~kola/kernel-replace: Add OS_ID recogntion~~ kola/kernel-replace: adapt for RHCOS and SCOS Jul 15, 2024

c4rt0 force-pushed the PR/kernel branch from 03d962c to 234036e Compare July 16, 2024 10:47

jbtrystram previously approved these changes Jul 16, 2024

View reviewed changes

tests/kola/rpm-ostree/kernel-replace Show resolved Hide resolved

c4rt0 dismissed jbtrystram’s stale review via fc70198 July 16, 2024 11:17

c4rt0 force-pushed the PR/kernel branch from 234036e to fc70198 Compare July 16, 2024 11:17

c4rt0 requested review from jbtrystram and jlebon July 16, 2024 11:25

jbtrystram approved these changes Jul 16, 2024

View reviewed changes

c4rt0 enabled auto-merge (rebase) July 16, 2024 11:56

jlebon approved these changes Jul 16, 2024

View reviewed changes

c4rt0 merged commit 721394b into coreos:testing-devel Jul 16, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kola/kernel-replace: adapt for RHCOS and SCOS #3048

kola/kernel-replace: adapt for RHCOS and SCOS #3048

c4rt0 commented Jul 2, 2024

jbtrystram left a comment •

edited

Loading

c4rt0 commented Jul 5, 2024 •

edited

Loading

jlebon commented Jul 5, 2024

c4rt0 commented Jul 9, 2024 •

edited

Loading

c4rt0 commented Jul 9, 2024 •

edited

Loading

c4rt0 commented Jul 10, 2024

jlebon left a comment

jlebon left a comment

jbtrystram left a comment

c4rt0 commented Jul 16, 2024

kola/kernel-replace: adapt for RHCOS and SCOS #3048

kola/kernel-replace: adapt for RHCOS and SCOS #3048

Conversation

c4rt0 commented Jul 2, 2024

jbtrystram left a comment • edited Loading

Choose a reason for hiding this comment

c4rt0 commented Jul 5, 2024 • edited Loading

jlebon commented Jul 5, 2024

c4rt0 commented Jul 9, 2024 • edited Loading

c4rt0 commented Jul 9, 2024 • edited Loading

c4rt0 commented Jul 10, 2024

jlebon left a comment

Choose a reason for hiding this comment

jlebon left a comment

Choose a reason for hiding this comment

jbtrystram left a comment

Choose a reason for hiding this comment

c4rt0 commented Jul 16, 2024

jbtrystram left a comment •

edited

Loading

c4rt0 commented Jul 5, 2024 •

edited

Loading

c4rt0 commented Jul 9, 2024 •

edited

Loading

c4rt0 commented Jul 9, 2024 •

edited

Loading